Apparatus and method for successive multi-frame image denoising

ABSTRACT

An apparatus and method for successive multi-frame image denoising are herein disclosed. According to one embodiment, an apparatus includes a Wiener filter configured to filter a frame of an image; and a frame delayer configured to feedback the filtered frame to the Wiener filter, wherein the Wiener filter is further configured to filter a subsequent frame of the image based on the filtered frame.

PRIORITY

This application claims priority under 35 U.S.C. § 119(e) to a U.S.Provisional Patent Applications filed on May 23, 2018 in the UnitedStates Patent and Trademark Office and assigned Ser. No. 62/675,377, theentire contents of which are incorporated herein by reference.

FIELD

The present disclosure relates generally to image processing, and moreparticularly, to an apparatus and method for successive multi-framedenoising.

BACKGROUND

Digital photography under low light conditions may be a challenging taskwhere a long exposure time (to collect more light) causes motion blurand a short exposure time (to avoid motion blur) increases the noiselevel. To overcome such challenges, a typical method is to use burstphotography and fuse the multiple frames to generate a sharp low-noiseimage. Multi-frame fusion, however, requires two major steps:registration to align the image frames (accounting for both global andlocal motion) and multi-frame merging (fusion) to enhance the quality.

SUMMARY

According to one embodiment, an apparatus is provided. The apparatusincludes a Wiener filter configured to filter a frame of an image; and aframe delayer configured to feedback the filtered frame to the Wienerfilter, wherein the Wiener filter is further configured to filter asubsequent frame of the image based on the filtered frame.

According to one embodiment, an apparatus is provided. The apparatusincludes a first subtractor including a first input to receive a frameof the image, a second input to receive a reference frame, and anoutput; an absolute value function block including an input connected tothe output of the first subtractor and an output; a second subtractorincluding a first input connected to the output of the absolute valuefunction block, a second input for receiving a first predeterminedvalue, and an output; and a maximum value divider function blockincluding an input connected to the output of the second subtractor andan output for outputting filter weights.

According to one embodiment, a method is provided. The method includesfiltering, by a Wiener filter, a frame of an image; and feeding back, bya frame delayer, the filtered frame to the Wiener filter; and filtering,by the Wiener filter, a subsequent frame of the image based on thefiltered frame.

According to one embodiment, a method is provided. The method includessubtracting, by a first subtractor, a reference from a frame of animage; determining, by an absolute value function block, an absolutevalue of an output of the first subtractor; subtracting, by a secondsubtractor, a first predetermined value from an output of the absolutevalue function block; and determining, by a maximum value dividerfunction block, a maximum value divider of an output of the secondsubtractor.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing detailed description, taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram of an apparatus for providing multi-framedenoising, according to one embodiment;

FIG. 2 is a block diagram of the merger of FIG. 1 for successive robustmerging, according to one embodiment;

FIG. 3 is a block diagram of the merger of FIG. 1 for differencesuccessive robust merging, according to one embodiment;

FIG. 4 is a block diagram of the merger of FIG. 1 for minimum meansquare error (MMSE) merging, according to one embodiment;

FIG. 5 is a block diagram of the merger of FIG. 1 for unbiased MMSE(U-MMSE) merging, according to one embodiment;

FIG. 6 is a block diagram of an apparatus for providing multi-framedenoising, according to one embodiment;

FIG. 7 is a block diagram of the merger of FIG. 6 for successivepairwise three dimensional (3D) merging, according to one embodiment;

FIG. 8 is a flowchart of a method of multi-frame denoising, according toone embodiment;

FIG. 9 is a flowchart of a method of multi-frame denoising, according toone embodiment;

FIG. 10 is a block diagram of an electronic device in a networkenvironment to which an apparatus and a method of the present disclosureis applied, according to one embodiment; and

FIG. 11 is a block diagram of a program to which an apparatus and amethod of the present disclosure is applied, according to oneembodiment.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT DISCLOSURE

Hereinafter, embodiments of the present disclosure are described indetail with reference to the accompanying drawings. The same elementswill be designated by the same reference numerals although they areshown in different drawings. In the following description, specificdetails such as detailed configurations and components are merelyprovided to assist with the overall understanding of the embodiments ofthe present disclosure. Therefore, it should be apparent to thoseskilled in the art that various changes and modifications of theembodiments described herein may be made without departing from thescope of the present disclosure.

In addition, descriptions of well-known functions and constructions areomitted for clarity and conciseness. The terms described below are termsdefined in consideration of the functions in the present disclosure, andmay be different according to users, intentions of the users, orcustoms. Therefore, the definitions of the terms should be determinedbased on the contents throughout this specification.

The present disclosure may have various modifications and variousembodiments, among which embodiments are described below in detail withreference to the accompanying drawings. However, the present disclosureis not limited to the embodiments, but includes all modifications,equivalents, and non-references within the scope of the presentdisclosure.

Although the terms including an ordinal number such as first, second,etc. may be used for describing various elements, the structuralelements are not restricted by the terms. The terms are only used todistinguish one element from another element. For example, withoutdeparting from the scope of the present disclosure, a first structuralelement may be referred to as a second structural element. Similarly,the second structural element may also be referred to as the firststructural element.

As used herein, the term “and/or” includes any and all combinations ofone or more associated items.

The terms used herein are merely used to describe various embodiments ofthe present disclosure but are not intended to limit the presentdisclosure. Singular forms are intended to include plural forms unlessthe context clearly indicates otherwise. In the present disclosure, itshould be understood that the terms “include” or “have” indicateexistence of a feature, a number, a step, an operation, a structuralelement, parts, or a combination thereof, and do not exclude theexistence or probability of the addition of one or more other features,numerals, steps, operations, structural elements, parts, or combinationsthereof.

Unless defined differently, all terms used herein have the same meaningsas those understood by a person skilled in the art to which the presentdisclosure belongs. Terms such as those defined in a generally useddictionary are to be interpreted to have the same meanings as thecontextual meanings in the relevant field of art, and are not to beinterpreted to have ideal or excessively formal meanings unless clearlydefined in the present disclosure.

According to one embodiment, an apparatus and method providesmulti-frame denoising also referred to as successive pairwise 3D mergingwhere a less noisy image is created with merging a burst of noisyimages. This reduces noise and is robust to residual alignment error(due to residual error in compensating both camera and scene motions).

Some level of alignment may already exist among the frames and focus maybe on a multi frame merging step. Furthermore, image frames may bealigned with some level of residual alignment error and artificialalignment errors may be introduced to evaluate the merging schemes inthe presence of alignment error.

In one embodiment, an apparatus selects one frame as a reference frame(either pre-determined or selected dynamically) and divides all framesinto small patches. The apparatus orders aligned non-reference patchesacross the burst based on their similarity with the correspondingreference patch, and successively merges the ordered patches intransform domain (e.g., discrete Fourier transform (DFT)). The apparatusperforms the merging successively (one patch at a time), where thenon-reference patches that are more similar to the reference patch aremerged first. The apparatus merges each non-reference patch with themost recently merged result) initialized with a reference patch in a 3Ddomain by applying Wiener filtering over 3D transform domain components,which improves peak signal to noise ratio (PSNR) and structuralsimilarity (SSIM) over synthetically generated noisy burst. Theapparatus and method performs better than other typical methodsincluding robust merging and V-BM4D.

In an embodiment, a robust merging method provides that exposure isconstant over all frames in a burst, which makes alignment more robust(there is no noise level difference between different frames of theburst). Second, raw Bayer images are merged rather than demosaiced redgreen blue (RGB) (or luminance (Y) and chrominance (U and V) (YUV))frames, which provides the benefit of both increased dynamic range and asimple noise model. Third, an FFT-based alignment algorithm and a hybridtwo dimensional (2D)/3D Wiener filter is used to denoise and mergeframes in a burst.

In an embodiment, both alignment and merging are performed per tile. Themerging method is based on a pairwise frequency-domain temporal filteroperating on the tiles of the input. Each tile in the reference ismerged with one tile taken from each of the non-reference frames. 16×16tiles are typically used from color planes of Bayer raw input, but forvery dark scenes, for which low-frequency noise can be objectionable,32×32 tiles are used.

In robust merging, each plane of the Bayer image is independentlymerged. Below, a merging method is disclosed in terms of single-channelimages which may be simply extended to the case of Bayer images whereeach plane is merged independently.

The robust merging method operates on image tiles in the spatialfrequency domain. For a given reference tile, a set of correspondingtiles are assembled across the burst, one per frame, and theirrespective 2D DFTs are computed as T_(z)(w), where w=(w_(x), w_(y))denotes spatial frequency, z is the frame index, and, without loss ofgenerality, frame 0 is taken to be the reference.

Robust merging applies frequency domain merging in Equations (1) asfollows:

$\begin{matrix}{{{\overset{\sim}{T}}_{0}(w)} = {{\frac{1}{N}{\sum\limits_{z = 0}^{N - 1}\lbrack {{T_{z}(w)} + {{A_{z}(w)}( {{T_{0}(w)} - {T_{z}(w)}} )}} \rbrack}} = {\frac{1}{N}{\sum\limits_{z = 0}^{N - 1}\lbrack {{T_{z}(w)} + {\frac{{{D_{z}(w)}}^{2}}{{{D_{z}(w)}}^{2} + {c\;\sigma^{2}}}{D_{z}(w)}}} \rbrack}}}} & (1)\end{matrix}$

A_(z) is a shrinkage operator that is defined as a variant of the Wienerfilter in Equation (2) as follows:

$\begin{matrix}{{A_{z}(w)} = \frac{{{D_{z}(w)}}^{2}}{{{D_{z}(w)}}^{2} + {c\;\sigma^{2}}}} & (2)\end{matrix}$where D_(z)(w)=T₀(w)−T_(z)(w), σ² denotes the noise variance, and c is adesign parameter that increases noise reduction at the expense of somerobustness.

The above pairwise temporal filter does not perform any spatialfiltering, so a spatial filtering is applied as a separatepost-processing step in the 2D DFT domain. Spatial filtering isperformed by applying a pointwise shrinkage operator of the same form asEquation (2) above to the spatial frequency coefficients, assuming thatall N frames are averaged perfectly and the estimated noise variance isupdated as σ²/N. An additional noise shaping filter in the form of{tilde over (σ)}=f(w)σ is also applied. In summary, the spatialfiltering of robust merging is in the form of Equation (3) as follows:

$\begin{matrix}{{{\hat{T}}_{0}(w)} = {\frac{{{\overset{\sim}{T}(w)}}^{2}}{{{\overset{\sim}{T}(w)}}^{2} + {c\frac{{f(w)}\sigma^{2}}{N}}}{\overset{\sim}{T}(w)}}} & (3)\end{matrix}$

Robust merging operates on tiles overlapped by half in each spatialdimension. By smoothly blending between overlapped tiles, visuallyobjectionable discontinuities may be avoided at tile boundaries.Additionally, a window function is applied to the tiles to avoid edgeartifacts when operating in the DFT domain. A modified raised cosinewindow is used, ½−½ cos(2π(x+½)/n) for x<n, and 0 otherwise.

According to one embodiment, the present apparatus and method providessuccessive robust merging as a modified version of the robust mergingmethod described above. In robust merging, the goal is to reject mergingof a non-reference frame whenever the difference between thecorresponding non-reference frame and the reference frame(D_(z)(w)=T₀(w)−T_(z)(w)) cannot be related to the noise.

In robust merging, non-reference frames are independently merged with areference frame. {circumflex over (R)}_(M) may be defined as the mergingresult after merging (M−1)-th non-reference frame. Therefore,{circumflex over (R)}_(N)(w)={tilde over (T)}₀(w) (as defined inEquation (1) above and the Equation (4) as follows) representsintermediate merging results of Equation (1) above:

$\begin{matrix}{{{\hat{R}}_{M}(w)} = \{ \begin{matrix}{T_{0}(w)} & \; & {M = 1} \\{\frac{1}{M}\sum\limits_{z = 0}^{M - 1}} & \lbrack {{T_{z}(w)} + {\frac{{{D_{z}(w)}}^{2}}{{{D_{z}(w)}}^{2} + {c\;\sigma^{2}}}{D_{z}(w)}}} \rbrack & {M > 1}\end{matrix} } & (4)\end{matrix}$

The goal of merging is to reduce the noise level, hence it is expectedto observe lower level of noise in {circumflex over (R)}_(M) for M>1 incompare with T₀. Assuming perfect alignment up to M-th frame, all framesare averaged perfectly and the noise level is reduced by

$\frac{1}{M}$after merging M frames. If there is access to the true noise-free imageof the scene R, a better merging strategy is to obtainD_(z)(w)=R(w)−T_(z)(w) as opposed to D_(z)(w)=T₀(w)−T_(z)(w) and rejectmerging the z-th non-reference frame whenever its difference withrespect to the true image R cannot be related to the noise. Typically,there is no access to the true image R; however, after merging M framesthere is access to {circumflex over (R)}_(M) as an estimate of R.

Therefore, an apparatus according to one embodiment may use {circumflexover (R)}_(M) in determining merging level of T_(M) ((M+1)-th frame.Frame indexing starts from 0, assuming that {circumflex over (R)}_(z)experiences noise level of

$\frac{\sigma^{2}}{z}.$This scheme may be referred to as successive robust merging, which maybe formulated as in Equation (5) as follows:

$\begin{matrix}{{{\hat{R}}_{M}(w)} = \{ \begin{matrix}{T_{0}(w)} & \; & {M = 1} \\{\frac{1}{M}\sum\limits_{z = 0}^{M - 1}} & {{T_{z}(w)} + {\frac{{{D_{z}^{S}(w)}}^{2}}{{{D_{z}^{S}(w)}}^{2} + c^{\;\frac{\sigma^{2}{({1 + \frac{1}{z}})}}{2}}}{D_{z}^{S}(w)}}} & {M > 1}\end{matrix} } & (5)\end{matrix}$where D_(z) ^(S)(w)={circumflex over (R)}_(z)(w)−T_(z)(w). Equation (5)may be equivalently written as Equation (6) as follows:

$\begin{matrix}{{{\hat{R}}_{M}(w)} = \{ \begin{matrix}{T_{0}(w)} & {M = 1} \\{\frac{1}{M}( {{( {M - 1} ){{\hat{R}}_{M - 1}(w)}} + {\frac{{{D_{M - 1}^{S}(w)}}^{2}}{{{D_{M - 1}^{S}(w)}}^{2} + {c\frac{\sigma^{2}( {1 + \frac{1}{M - 1}} )}{2}}}{D_{M - 1}^{S}(w)}}} )} & {M > 1}\end{matrix} } & (6)\end{matrix}$

According to one embodiment, an apparatus and a method providesdifferential successive robust merging, where {circumflex over(R)}_(M)(w) is only used to update A_(z)(w)'s which arrives at Equation(7) as follows:

$\begin{matrix}{{{\hat{R}}_{M}(w)} = \{ \begin{matrix}{T_{0}(w)} & {M = 1} \\{\frac{1}{M}{\sum\limits_{z = 0}^{M - 1}\lbrack {{T_{z}(w)} + {\frac{{{D_{z}^{S}(w)}}^{2}}{{{D_{z}^{S}(w)}}^{2} + c^{\;\frac{\sigma^{2}{({1 + \frac{1}{z}})}}{2}}}{D_{z}(w)}}} \rbrack}} & {M > 1}\end{matrix} } & (7)\end{matrix}$where D_(z) ^(s)(w)={circumflex over (R)}_(z)(w)−T_(z)(w) andD_(z)(w)=T₀(w)−T_(z)(w) and the following may be equivalently written asEquation (8) as follows:

$\begin{matrix}{{{\hat{R}}_{M}(w)} = \{ \begin{matrix}{T_{0}(w)} & {M = 1} \\{\frac{1}{M}( {{( {M - 1} ){{\hat{R}}_{M - 1}(w)}} + {\frac{{{D_{M - 1}^{S}(w)}}^{2}}{{{D_{M - 1}^{S}(w)}}^{2} + {c\frac{\sigma^{2}( {1 + \frac{1}{M - 1}} )}{2}}}{D_{M - 1}(w)}}} )} & {M > 1}\end{matrix} } & (8)\end{matrix}$

According to one embodiment, a burst of N noisy images with I_(k)(x,y)denotes the k-th image frame k∈{0, . . . , N−1} and (x,y) denotes the 2Dspatial pixel position. Patches are overlapped by half in each spatialdimension with size of 16×16 and filtered using a modified raised cosinewindow, ½−½ cos (2π(x+½)/n) for x<n, and 0 otherwise. Atwo-dimensional-discrete Fourier transform (2D-DFT) of a consideredpatch (weighted by the aforementioned raised cosine window) of the k-thframe is denoted as F_(k)(w), where w=(w_(x), w_(y)), denotes 2D spatialfrequency and the patch index is dropped for the sake of simplicity.

According to one embodiment, an apparatus and method providesmulti-frame merging in particular, successive pairwise 3D merging.Residual alignment error may exist among the frames. In successivepairwise 3D merging, the present apparatus successively mergesnon-reference frames where each non-reference frame is merged with themost recent merged result (initialized with a reference patch). Thealigned non-reference patches across the burst are ordered based ontheir similarity with the corresponding reference patch, thensuccessively merged in a transform domain. The apparatus performsmerging successively (one patch at a time), where the non-referencepatches more similar to the reference patch are merged first. Theapparatus performs merging in a transform domain using a two-steptransformation.

In a first step transformation, the present apparatus obtainstwo-dimensional transformation (e.g., discrete Fourier transforms2D-DFTs) of the reference and non-reference frames. In a second steptransformation, the present apparatus performs a simple 2 pointone-dimensional-DFT (1D-DFT) in temporal direction over the pair ofaligned frequency components of the most recent merged results and theavailable non-reference frame.

The apparatus applies Wiener filtering over the 3D transformationresults, then converts the filtered 3D transformation results back to 2Ddomain (simple 2 point inverse DFT) to update the most recent mergedresults. The apparatus initializes the merging with Wiener filtering ofthe reference frame in 2D transform domain. For example, the successivepairwise merging scheme may be formulated as in Equations (9) and (10)as follows:

$\begin{matrix}{{{\hat{R}}_{M}(w)} = \{ \begin{matrix}{\frac{{{T_{0}(w)}}^{2}}{{{T_{0}(w)}}^{2} + {c\;\sigma^{2}}}{T_{0}(w)}} & {M = 1} \\{\frac{1}{2}( {{G_{M}^{+}( {{{\hat{R}}_{M - 1}(w)} + {T_{M - 1}(w)}} )} + {G_{M}^{-}( {{{\hat{R}}_{M - 1}(w)} - {T_{M - 1}(w)}} )}} )} & {M > 1}\end{matrix} } & (9) \\{{{\hat{R}}_{M}(w)} = \{ \begin{matrix}{F_{0}(w)} & {M = 1} \\\frac{{G_{M}^{+}( {{{\hat{R}}_{M - 1}(w)} + {F_{M - 1}(w)}} )} + {G_{M}^{-}( {{{\hat{R}}_{M - 1}(w)} - {F_{M - 1}(w)}} )}}{2} & {M > 1}\end{matrix} } & (10)\end{matrix}$where {circumflex over (R)}_(M) represent the most recent merge resultafter merging M frames (merging M−1 non-reference frames to thereference frame), and Equations (11) and (12) are as follows:

$\begin{matrix}{{G_{M}^{+}(w)} = \frac{{{{{\hat{R}}_{M - 1}(w)} + {T_{M - 1}(w)}}}^{2}}{{{{{\hat{R}}_{M - 1}(w)} + {T_{M - 1}(w)}}}^{2} + {c( {\sigma_{M - 1}^{2} + \sigma^{2}} )}}} & (11) \\{{G_{M}^{-}(w)} = \frac{{{{{\hat{R}}_{M - 1}(w)} + {T_{M - 1}(w)}}}^{2}}{{{{{\hat{R}}_{M - 1}(w)} - {T_{M - 1}(w)}}}^{2} + {c( {\sigma_{M - 1}^{2} + \sigma^{2}} )}}} & (12)\end{matrix}$where σ_(M) ²=¼[(G_(M-1) ⁺(w)+G_(M-1) ⁻(w))²σ_(M-1) ⁺−G_(M-1) ⁺(w))²σ²]with σ₀ ²=σ².

In successive merging, the order of the merging may impact the qualityof the final merging results. Different metrics may be used in orderingthe frames, for example, sharpness (sharpest frames selected as thereference and the sharper frames are merged first) or matching score ofthe non-reference frames with respect to the reference frame where theframes with higher matching score are merged first.

According to one embodiment, the apparatus provides merging based on anMMSE estimation principle in the frequency domain. A motivation inconsidering MMSE estimation in frequency domain (instead of spatialdomain) is the sparsity property of the natural images in frequencydomain enabling us to obtain MMSE estimation independently per frequencycomponent. The apparatus applies MMSE estimation in a transform domain(frequency domain) that considers each frequency componentindependently.

As described above in reference to the robust merging method, overlappedpatches are merged in the frequency domain, where T_(z)(w) is defined asa frequency domain representation of the z-th frame tile and w=(w_(x),w_(y)) denotes spatial frequency.

R is defined as the true signal (e.g., the original noise free image ofthe scene), T₀(w) is considered as the noisy reference frame andI_(z)(w) is defined as an alignment error at z-th frame with respect tothe reference frame, i.e., I₀(w)=0. For noisy and potentially misalignedblock of the z-th non-reference frame T_(z)(w), Equation (13) is asfollows:T _(z)(w)=R(w)+I _(z)(w)+N(w)  (13)where R is the true image and N_(z)(w) is independent and identicallydistributed (i.i.d.) zero mean Gaussian noise with variance σ_(z) ²;T₀(w)=R(w)+N₀(w).

According to one embodiment, the apparatus performs merging as anestimation problem with the goal of estimating the image R out of Nobservations T_(z)(w)'s. For I₀(w)=0, frame T₀ acts as the referenceframe. Considering a burst of N frames, and for a given spatialfrequency w, a vector T (w)=[T₀(w), T₁(w), . . . , T_(N-1)(w)]^(T) isformed that may be formulated as in Equation (14) as follows:

$\begin{matrix}{{T(w)} = {\begin{bmatrix}{T_{0}(w)} \\{T_{1}(w)} \\\vdots \\{T_{N - 1}(w)}\end{bmatrix} = {{\begin{bmatrix}1 \\1 \\\vdots \\1\end{bmatrix}{R(w)}} + \begin{bmatrix}0 \\{I_{1}(w)} \\\vdots \\{I_{N - 1}(w)}\end{bmatrix} + {N(w)}}}} & (14)\end{matrix}$

If μ(w)=E{R(w)} and P(w) as true image power at spatial frequency w,i.e., P(w)=E{|R(w)−μ(w)|²} and I_(z)(w)'s are i.i.d. Gaussian variableswith E{I₂(w)}=0, for MMSE estimation of R(w), Equation (15) is asfollows:

$\begin{matrix}{{\hat{R}(w)} = {{\mu(w)} + {{{P(w)}\begin{bmatrix}1 & 1 & \ldots & 1\end{bmatrix}}( {{{{P(w)}\begin{bmatrix}1 \\1 \\\vdots \\1\end{bmatrix}}\begin{bmatrix}1 & 1 & \ldots & 1\end{bmatrix}} + \begin{bmatrix}\sigma_{0}^{2} & 0 & \ldots & 0 \\0 & {\sigma_{1}^{2} + {E\{ {{I_{1}(w)}}^{2} \}}} & \ldots & 0 \\\vdots & \vdots & \ddots & \vdots \\0 & 0 & \ldots & {\sigma_{N - 1}^{2} + {E\{ {{I_{N - 1}(w)}}^{2} \}}}\end{bmatrix}} )^{- 1}( {{T(w)} - {{\mu(w)}1_{N}}} )}}} & (15)\end{matrix}$

From Equation (15), Equation (16) may be obtained as follows:

$\begin{matrix}{{{\hat{R}}_{MMSE}(w)} = {{\mu(w)} + {\frac{1}{1 + {\sum\limits_{z = 0}^{N - 1}\frac{P(w)}{\sigma_{z}^{2} + {E\{ {{I_{z}(w)}}^{2} \}}}}}{\sum\limits_{z = 0}^{N - 1}{\frac{P(w)}{\sigma_{z}^{2} + {E\{ {{I_{z}(w)}}^{2} \}}}( {{T_{z}(w)} - {\mu(w)}} )}}}}} & (16)\end{matrix}$

To obtain the MMSE results, σ+E{|I_(z)(w)|²} and P(w) are required whichare unknown. For σ_(z) ²+E{|I_(z)(w)|²}, the instantaneous power may beused, i.e., as in Equation (17) as follows:{|I _(z)(w)|²}+σ_(z) ²=σ_(z) ²+max(0,|T _(z)(w)−T ₀(w)|²−σ₀ ²−σ_(z)²  (17)

For P(w), since there is access to multiple frames, P(w) may bedetermined as in Equation (18) as follows:

$\begin{matrix}{{P(w)} = {\frac{1}{N}{\sum\limits_{z = 0}^{N - 1}{{\max( {0,{{{{T_{z}(w)} - {\mu(w)}}}^{2} - ( {{E\{ {{I_{z}(w)}}^{2} \}} + \sigma_{z}^{2}} )}} )}.}}}} & (18)\end{matrix}$

Since E{{circumflex over (R)}(w)|R(w)}≠R(w), the above estimator isbiased. Therefore, an unbiased MMSE estimate of R(w) may be obtained asin Equation (19) as follows:

$\begin{matrix}{{{\hat{R}}_{UMMSE}(w)} = {\frac{1}{\sum\limits_{z = 0}^{N - 1}\frac{1}{\sigma_{z}^{2} + {E\{ {{I_{z}(w)}}^{2} \}}}}{\sum\limits_{z = 0}^{N - 1}{\frac{1}{\sigma_{z}^{2} + {E\{ {{I_{z}(w)}}^{2} \}}}{T_{z}(w)}}}}} & (19)\end{matrix}$where only an estimate of E{|I_(z)(w)|²}+σ_(z) ² is needed to obtain{circumflex over (R)}_(UMMSE)(w).μ(w)=E{R(w)} is also needed in the above calculations. μ(w) may becalculated in terms of the mean of the patch in spatial domain. Bydefining J(x,y) as image intensity at (x,y) spatial location, andassuming E{J(x,y)}=u, Equation (20) is obtained as follows:

$\begin{matrix}{{\mu(w)} = {{E\{ {{DFT}( {{G.} \star J} )} \}} = {{\frac{1}{M}{\sum\limits_{z = 0}^{M - 1}{\sum\limits_{z = 0}^{M - 1}{{G( {x,y} )}E\{ {J( {x,y} )} \} e^{- \frac{2\pi\;{j{({{w_{x}x} + {w_{y}y}})}}}{M}}}}}} = {{uDFT}(G)}}}} & (20)\end{matrix}$where G represents the windowing matrix and “.*” represents element wisematrix multiplication.

In the present disclosure, five sets of multi-frame denoisingapparatuses and methods are described (i.e., successive robust merging(which is combined with robust merging), difference successive robustmerging, successive pairwise 3D merging (which applies a pairwise Wienerfiltering in a 3D transform domain), MMSE merging, and unbiased MMSEmerging).

A burst of N noisy images with I_(k)(x,y) denoting the k-th image frame,k∈(0, . . . , N−1) and (x,y) denoting the 2D spatial pixel position areconsidered. For notational convenience, the color channel index isomitted. However, the present disclosure may include color channels.

M×M image patches are extracted and a 2D transformation (e.g., 2D-DFT or2D wavelet) is applied over the patches and the image patch in atransform domain is denoted as F_(k)(w_(x),w_(y)) where w=(w_(x), w_(y))denotes 2D spatial frequency. For example, image patches may be selectedby half overlap in each spatial dimension with size of M×M and filteredusing a modified raised cosine window, ½−½ cos (2π(x+½)/n) for x<n, and0 otherwise.

In an embodiment, some level of alignment is obtained among the frames,where the focus is on multi frame merging in the presence of residualalignment error.

FIG. 1 is a block diagram of an exemplary apparatus 100 for providingmulti-frame image denoising, according to one embodiment.

Referring to FIG. 1, the apparatus 100 includes an image registrator101, a patch extractor 103, a 2D transformer 105, a merger 107, aspatial denoiser 109, and a 2D inverse transformer and patch combiner111.

The image registrator 101 includes an input 113 for receiving a signalI_(k) and an output 115. The patch extractor 103 includes an inputconnected to the output 115 of the image registrator 101 and an output117. The output 117 of the patch extractor 103 may output an M×M sizedpatch, where M is a rational number. The 2D transformer 105 includes aninput connected to the output 117 of the patch extractor 103. The output119 of the 2D transformer 105 may be frame w (e.g., F_(z)(w)) of theinput signal. The merger 107 includes an input connected to the output119 of the 2D transformer 105 and an output 121. The output 121 of themerger 107 may provide an intermediate estimate {circumflex over (R)}(w)of size M×M. The spatial denoiser 109 includes an input connected to theoutput 121 of the merger 107 and an output 123. The output 123 of thespatial denoiser 109 may provide a post-processed estimate {circumflexover (R)}(w) of size M×M. The 2D inverse transformer and patch combiner111 includes an input connected to the output 123 of the spatialdenoiser 109 and an output 125 outputting a multi-frame image denoisedsignal Î₀.

Image patches are selected by half overlap in each spatial dimensionwith size of M×M and filtered using a modified raised cosine window, ½−½cos (2π(x+½)/n) for x<n, and 0 otherwise. 2D-DFT is used as a 2Dtransformation. Frame 0 is selected as a reference frame. {tilde over(R)}(w) is obtained as in Equation (21) as follows:

$\begin{matrix}{{\overset{\sim}{R}(w)} = {{\frac{1}{N}{\sum\limits_{z = 0}^{N - 1}\lbrack {{F_{z}(w)} + {{A_{z}(w)}( {{F_{0}(w)} - {F_{z}(w)}} )}} \rbrack}} = {\frac{1}{N}{\sum\limits_{z = 0}^{N - 1}\lbrack {{F_{z}(w)} + {\frac{{{D_{z}(w)}}^{2}}{{{D_{z}(w)}}^{2} + {c\;\sigma^{2}}}{D_{z}(w)}}} \rbrack}}}} & (21)\end{matrix}$

For a given frequency, A_(z)(w) controls the degree to which z-thnon-reference frame is merged into the final result versus falling backto the reference frame. Equation (21) can be rewritten as in Equation(22) as follows:

$\begin{matrix}{{\overset{\sim}{R}(w)} = {\frac{1}{N}{\sum\limits_{z = 0}^{N - 1}\lbrack {{( {1 - {A_{z}(w)}} ){F_{z}(w)}} + {{A_{z}(w)}{F_{0}(w)}}} \rbrack}}} & (22)\end{matrix}$

The above pairwise temporal filter does not perform any spatialfiltering. Thus, a spatial filtering is applied as a separatepost-processing step in the 2D DFT domain. Spatial filtering isperformed by applying a pointwise shrinkage operator. An additionalnoise shaping filter in the form of {tilde over (σ)}=f(w)σ is alsoapplied. The spatial filtering of robust merging is as in Equation (23)as follows:

$\begin{matrix}{{{\hat{F}}_{0}(w)} = {\frac{{{\overset{\sim}{F}(w)}}^{2}}{{{\overset{\sim}{F}(w)}}^{2} + {c\frac{\;{f(w)\sigma^{2}}}{N}}}{\overset{\sim}{F}(w)}}} & (23)\end{matrix}$

FIG. 2 is a block diagram of an exemplary apparatus for providingsuccessive robust merging, according to one embodiment.

Referring to FIG. 2, the present apparatus includes a merger 200 havinga subtractor 201, a Wiener filter 203, a summation function block 205, amultiplier 207, and a frame delayer 209.

The subtractor 201 includes a first input 211 for receiving F_(z)(w), asecond input connected to the output 223 of the frame delayer 209 forreceiving {circumflex over (R)}_(z)(w), and an output 213 for outputting{circumflex over (R)}_(z)(w)−F_(z)(w). The Wiener filter 203 includes afirst input connected to the output 213 of the subtractor 201, a secondinput connected to the output 213 of the subtractor 201, and an output215. The summation function block 205 includes a first input connectedto the output 215 of the Wiener filter 203, a second input 211 forreceiving F_(z)(w), and an output 217 for outputting {circumflex over(R)}(w). The multiplier 207 includes a first input connected to theoutput 217 of the summation function block 205, a second input 219 forreceiving 1/z, and an output 221. The frame delayer 209 includes aninput connected to the output 221 of the multiplier 207 and an output223 for outputting {circumflex over (R)}_(z)(w).

The goal of merging is to reject merging of an non-reference framewhenever a difference between a corresponding non-reference frame and areference frame cannot be related to noise.

However, after merging z frames we have access to {circumflex over(R)}_(z) as an estimate of R. Therefore, {circumflex over (R)}_(z) canbe used in determining merging level of F_(z), assuming that {circumflexover (R)}_(z) experiences noise level of

$\frac{\sigma^{2}}{z}.$This scheme is referred to as successive robust merging, which can beformulated as in Equation (24) as follows:

$\begin{matrix}{{{\hat{R}}_{z + 1}(w)} = \{ \begin{matrix}{F_{0}(w)} & {z = 0} \\{{\frac{1}{z + 1}{\sum\limits_{m = 0}^{z}{F_{m}(w)}}} + {\frac{{{D_{m}^{S}(w)}}^{2}}{{{D_{m}^{S}(w)}}^{2} + {c\;\frac{\sigma^{2}( {1 + \frac{1}{m}} )}{2}}}{D_{m}^{S}(w)}}} & {z \geq 1}\end{matrix} } & (24)\end{matrix}$where D_(m) ^(s)(w)={circumflex over (R)}_(m)(w)−F_(m)(w), which may beequivalently formulated as in Equation (25) as follows:

$\begin{matrix}{{{\hat{R}}_{z + 1}(w)} = \{ \begin{matrix}{F_{0}(w)} & {z = 0} \\{\frac{1}{z + 1}( {{z\;{{\hat{R}}_{z}(w)}} + {\frac{{{D_{z}^{S}(w)}}^{2}}{{{D_{m}^{S}(w)}}^{2} + {c\;\frac{\sigma^{2}( {1 + \frac{1}{z}} )}{2}}}{D_{z}^{S}(w)}}} )} & {z \geq 1}\end{matrix} } & (25)\end{matrix}$

FIG. 3 is a block diagram of an exemplary apparatus for providingdifference successive robust merging, according to one embodiment.

Referring to FIG. 3, the present apparatus includes a merger 300 havinga first subtractor 301, a second subtractor 303, a Wiener filter 305, asummation function block 307, a multiplier 309, and a frame delayer 311.

The first subtractor 301 includes a first input 313 for receivingF_(z)(w), a second input 315 for receiving F₀(w), and an output 317 foroutputting F₀(w)−F_(z)(w). The second subtractor 303 includes a firstinput 313 for receiving F_(z)(w), a second input connected to an output325 of the frame delayer 311 for receiving {circumflex over (R)}_(z)(w),and an output 329 for outputting {circumflex over (R)}_(z)(w)−F_(z)(w).The Wiener filter 305 includes a first input connected to the output 317of the first subtractor 301, a second input connected to the output 329of the second subtractor 303, and an output 319. The summation functionblock 307 includes a first input connected to the output 319 of theWiener filter 305, a second input 313 for receiving F_(z)(w), and anoutput 321 for outputting {circumflex over (R)}(w). The multiplier 309includes a first input connected to the output 321 of the summationfunction block 307, a second input 323 for receiving 1/z, and an output325. The frame delayer 311 includes an input connected to the output 325of the multiplier 309 and an output 327 for outputting {circumflex over(R)}_(z)(w).

{circumflex over (R)}_(z)(w) may be used for updating filtering weightsand may be formulated as in Equation (26) as follows:

$\begin{matrix}{{{\hat{R}}_{z + 1}(w)} = \{ \begin{matrix}{F_{0}(w)} & {z = 0} \\{\frac{1}{z + 1}{\sum\limits_{m = 0}^{z}\lbrack {{F_{m}(w)} + {\frac{{{D_{m}^{S}(w)}}^{2}}{{{D_{m}^{S}(w)}}^{2} + {c\;\frac{\sigma^{2}( {1 + \frac{1}{m}} )}{2}}}{D_{m}(w)}}}\; \rbrack}} & {z \geq 1}\end{matrix} } & (26)\end{matrix}$where D_(m) ^(S)(w)={circumflex over (R)}_(m)(w)−F_(m)(w) andD_(m)(w)=F₀(w)−F_(m)(w) and where Equation (26) above may beequivalently formulated as Equation (27) as follows:

$\begin{matrix}{{{\hat{R}}_{z + 1}(w)} = \{ \begin{matrix}{{F_{0}(w)}\mspace{11mu}} & {\;{z = 0}} \\{\frac{1}{z + 1}( {{z\;{{\hat{R}}_{z}(w)}} + {\frac{{{D_{m}^{S}(w)}}^{2}}{{{D_{m}^{S}(w)}}^{2} + {c\;\frac{\sigma^{2}( {1 + \frac{1}{m}} )}{2}}}{D_{z}(w)}}} )} & {z \geq 1}\end{matrix} } & (27)\end{matrix}$

FIG. 4 is a block diagram of a merger 300 for minimum mean square error(MMSE) merging, according to one embodiment.

Referring to FIG. 4, the merger 300 includes a first subtractor 401, afirst multiplier 403, a first summation function block 405, a secondmultiplier 407, a divider 409, a first adder 411, a second subtractor413, an absolute value function block 415, a third subtractor 417, amaximum value divider 419, a second summation function block 421, athird multiplier 423, and a second adder 425.

The first subtractor 401 includes a first input 427 for receivingF_(z)(w), a second input 429 for receiving μ(w), and an output 431 foroutputting μ(w)−F_(z)(w). The first multiplier 403 includes a firstinput connected to the output 431 of the first subtractor 401, a secondinput connected to an output 457 of the maximum value divider 419 forreceiving filter weights, and an output 433. The first summationfunction block 405 includes an input connected to the output 433 of thefirst multiplier 403, and an output 435. The second multiplier 407includes a first input connected to the output 435 of the firstsummation function block 407, a second input 437 for receiving P (w),and an output 439. The divider 409 includes a first input connected tothe output 439 of the second multiplier 407, a second input connected toan output 467 of the second adder 425, and an output 441. The firstadder 411 includes a first input connected to the output 441 of thedivider 409, a second input 443 for receiving μ(w), and an output 445The second subtractor 413 includes a first input 427 for receivingF_(z)(w), a second input 447 for receiving F₀(w), and an output 449 foroutputting F_(k)(w)−F₀(w). The absolute value function block 415includes an input connected to the output 449 of the second subtractor413 and an output 451. The third subtractor 417 includes a first inputconnected to the output 451 of the absolute value function block 415, asecond input 453 for receiving σ₀ ^(z), and an output 455. The maximumvalue divider function block 419 includes an input connected to theoutput 455 of the third subtractor 417 and an output 457 for outputtingfilter weights. The second summation function block 421 includes aninput connected to the output 457 of the maximum value divider 419, andan output 459. The third multiplier 423 includes a first input connectedto the output 459 of the second summation function block 421, a secondinput 461 for receiving P (w), and an output 463. The second adder 425includes a first input connected to the output 463 of the thirdmultiplier 423, a second input 465 for receiving a value 1, and anoutput 467.

R may be a true signal (e.g., an original noise free image of a scene),where F₀(w) is a noisy reference frame and e_(z)(w) is an alignmenterror at a z-th frame with respect to the reference frame, i.e.,e₀(w)=0. For a noisy and potentially misaligned block of the z-thnon-reference frame F_(z)(w), Equation (28) is a follows:F _(z)(w)=R(w)+e _(z)(w)+N _(z)(w)  (28)where R is the true image and N_(z)(w) is i.i.d. zero mean Gaussiannoise with variance σ_(z) ². Thus, F₀(w)=R(w)+N₀(w).

The merging solution may be considered as an estimation problem with thegoal of estimating the image R out of N observations F_(z)(w)'s. Byassuming e₀(w)=0, frame F₀ acts as the reference frame. Considering aburst of N frames, and for a given spatial frequency w, a vectorF(w)=[F₀(w), F₁(w), . . . , F_(N-1)(w)]^(T) may be formulated as inEquation (29) as follows:

$\begin{matrix}{{F(w)} = {\begin{bmatrix}{F_{0}(w)} \\{F_{1}(w)} \\\vdots \\{F_{N - 1}(w)}\end{bmatrix} = {{\begin{bmatrix}1 \\1 \\\vdots \\1\end{bmatrix}{R(w)}} + \begin{bmatrix}0 \\{e_{1}(w)} \\\vdots \\{e_{N - 1}(w)}\end{bmatrix} + {N(w)}}}} & (29)\end{matrix}$

If μ(w)=E{R(w)} and P(w) is a true image power at spatial frequency w,i.e., P(w)=E {|R(w)−μ(w)|²} and e_(z)(w)'s are i.i.d Gaussian variableswith E{e_(z)(w)}=0, for MMSE estimation of R(w), Equation (30) is asfollows:

$\begin{matrix}{{\hat{R}(w)} = {{\mu(w)} + {{{P(w)}\lbrack {1\mspace{14mu} 1\mspace{14mu}\ldots\mspace{14mu} 1} \rbrack}( {{{{P(w)}\begin{bmatrix}1 \\1 \\\vdots \\1\end{bmatrix}}\lbrack {1\mspace{14mu} 1\mspace{14mu}\ldots\mspace{14mu} 1} \rbrack} + \begin{bmatrix}\sigma_{0}^{2} & 0 & \ldots & 0 \\0 & {\sigma_{1}^{2} + {E\{ {{e_{1}(w)}}^{2} \}}} & \ldots & 0 \\\vdots & \vdots & \ddots & \vdots \\0 & 0 & \ldots & {\sigma_{N - 1}^{2} + {E\{ {{e_{N - 1}(w)}}^{2} \}}}\end{bmatrix}} )^{- 1}( {{F(w)} - {{\mu(w)}1_{N}}} )}}} & (30)\end{matrix}$

Equation (30) above may be formulated as Equation (31) as follows:

$\begin{matrix}{{{\hat{R}}_{MMSE}(w)} = {{\mu(w)} + {\frac{1}{1 + {\sum\limits_{z = 0}^{N - 1}\frac{P(w)}{\sigma_{z}^{2} + {E\{ {{e_{z}(w)}}^{2} \}}}}}{\sum\limits_{z = 0}^{N - 1}{\frac{P(w)}{\sigma_{z}^{2} + {E\{ {{e_{z}(w)}}^{2} \}}}( {{F_{z}(w)} - {\mu(w)}} )}}}}} & (31)\end{matrix}$

To obtain the MMSE results, σ_(z) ²+E{|e_(z)(w)|²} and P(w) arerequired, which are unknown. For σ_(z) ²+E{|e_(z)(w)|²}, theinstantaneous power is as in Equation (32) as follows:E{|e _(z)(w)|²}+σ_(z) ²=σ_(z) ²+max(0,|F _(z)(w)−F ₀(w)|²−σ₀ ²−σ_(z)²)  (32)

Since multiple frames are available, P(w) is calculated from Equation(33) as follows:

$\begin{matrix}{{P(w)} = {\frac{1}{N}{\sum\limits_{z = 0}^{N - 1}{\max( {0,{{{{F_{z}(w)} - {\mu(w)}}}^{2} - ( {{E\{ {{e_{z}(w)}}^{2} \}} + \sigma_{z}^{2}} )}} )}}}} & (33)\end{matrix}$

Since E{{circumflex over (R)}(w)|R(w)}≠R(w), the MMSE merging estimateof R (w) is biased.

FIG. 5 is a block diagram of a merger 500 for U-MMSE merging, accordingto one embodiment.

Referring to FIG. 5, the merger 500 includes a multiplier 501, a firstsummation function block 503, a divider 505, a first subtractor 507, anabsolute value function block 509, a second subtractor 511, a maximumvalue divider 513, and a second summation function block 515.

The multiplier 501 includes a first input 517 for receiving F_(z)(w), asecond input connected to an output 535 of the maximum value divider 513for receiving filter weights, and an output 519. The first summationfunction block 503 includes an input connected to the output 519 of themultiplier 501, and an output 521. The divider 505 includes a firstinput connected to the output 521 of the first summation function block503, a second input connected to an output 537 of the second summationfunction block 515, and an output 523. The first subtractor 507 includesa first input 517 for receiving F_(z)(w), a second input 525 forreceiving F₀(w), and an output 527 for outputting F_(z)(w)−F₀(w). Theabsolute value function block 509 includes an input connected to theoutput 527 of the first subtractor 507 and an output 531. The secondsubtractor 511 includes a first input connected to the output 531 of theabsolute value function block 509, a second input 531 for receiving σ₀², and an output 533. The maximum value divider function block 513includes an input connected to the output 533 of the second subtractor511 and an output 535 for outputting filter weights. The secondsummation function block 515 includes an input connected to the output535 of the maximum value divider 513, and an output 537.

An unbiased MMSE estimate of R (w) is formulated in Equation (34) asfollows:

$\begin{matrix}{{{\hat{R}}_{UMMSE}(w)} = {\frac{1}{\sum\limits_{z = 0}^{N - 1}\frac{1}{\sigma_{2}^{2} + {E\{ {{e_{z}(w)}}^{2} \}}}}{\sum\limits_{z = 0}^{N - 1}{\frac{1}{\sigma_{z}^{2} + {E\{ {{e_{z}(w)}}^{2} \}}}{F_{z}(w)}}}}} & (34)\end{matrix}$where only an estimate of E{|e_(z)(w)|²}+σ_(z) ² is needed to obtain{circumflex over (R)}_(UMMSE)(w).

FIG. 6 is a block diagram of another exemplary apparatus for providingmulti-frame image denoising, according to one embodiment.

Referring to FIG. 7, the apparatus 600 includes an image registrator601, a patch extractor 603, a 2D transformer 605, a merger 607, and a 2Dinverse transformer and patch combiner 609.

The image registrator 601 includes an input 611 for receiving a signalI_(k) and an output 613. The patch extractor 603 includes an inputconnected to the output 613 of the image registrator 601 and an output615. The 2D transformer 605 includes an input connected to the output615 of the patch extractor 603. The merger 607 includes an inputconnected to the output 617 of the 2D transformer 605 and an output 619.The 2D inverse transformer and patch combiner 609 includes an inputconnected to the output 619 of the merger 607 and an output 621outputting a multi-frame image denoised signal Î₀.

FIG. 7 is a block diagram of the merger 607 of FIG. 7 for successivepairwise 3D merging, according to one embodiment.

Referring to FIG. 8, the merger 607 includes a subtractor 701, a firstadder 703, a first Wiener filter 705, a second Wiener filter 707, asecond adder 709, a multiplier 711, and a frame delayer 713.

The subtractor 701 includes a first input connected to the input 617 ofthe merger 607 for receiving F_(z)(w), a second input connected to anoutput 727 of the frame delayer 713 for receiving {circumflex over(R)}_(z)(w), and an output 715 for outputting {circumflex over(R)}_(z)(w)−F₀(w). The first adder 703 includes a first input connectedto the input 617 of the merger 607 for receiving F_(z)(w), a secondinput connected to the output 727 of the frame delayer 713 for receiving{circumflex over (R)}_(z)(w), and an output 719 for outputting{circumflex over (R)}_(z)(w)+F₀(w). The second Wiener filter 707includes a first input connected to the output 719 of the first adder703, a second input connected to the output 719 of the first adder 703,and an output 721. The second adder 709 includes a first input connectedto the output 721 of the second Wiener filter 707, a second inputconnected to the output 717 of the first Wiener filter 705, and anoutput connected to the output 619 of the merger 607. The multiplier 711includes a first input connected to the output 619 of the second adder709, a second input 723 for receiving a value 0.5, and an output 725.The frame delayer 713 includes an input connected to the output 725 ofthe multiplier 711 and an output 727.

Non-reference frames are successively merged where each non-referenceframe is merged with the most recent merge result and the merging isperformed in transform domain using a two-step transformation. In afirst step, a 2D transformation of a reference frame and non-referenceframes are obtained. A second step transformation is a 2 point 1D-DFT intemporal direction over the pair of aligned frequency components of themost recent merge results and the available non-reference frame. TheWiener filtering is applied over the 3D transformation results, then itis converted back to 2D domain (simple 2 point inverse DFT) to updatethe most recent merge results. Merging is initialized with the referenceframe. The successive pairwise merging scheme may be formulated as inEquations (35), (36), and (37) as follows:

$\begin{matrix}{{{\hat{R}}_{z + 1}(w)} = \{ \begin{matrix}{F_{0}(w)} & {z = 0} \\{{\frac{1}{2}\begin{pmatrix}{{G_{z + 1}^{+}( {{{\hat{R}}_{z}(w)} + {F_{2}(w)}} )} +} \\{G_{z + 1}^{-}( {{{\hat{R}}_{z}(w)} - {F_{z}(w)}} )}\end{pmatrix}}\;} & {z \geq 1}\end{matrix} } & (35) \\{{G_{z + 1}^{+}(w)} = \frac{{{{{\hat{R}}_{z}(w)} + {F_{z}(w)}}}^{2}}{{{{{\hat{R}}_{z}(w)} + {F_{z}(w)}}}^{2} + {c( {\sigma_{z}^{2} + \sigma^{2}} )}}} & (36) \\{{G_{z + 1}^{-}(w)} = \frac{{{{{\hat{R}}_{z}(w)} - {F_{z}(w)}}}^{2}}{{{{{\hat{R}}_{z}(w)} - {F_{z}(w)}}}^{2} + {c( {\sigma_{z}^{2} + \sigma^{2}} )}}} & (37)\end{matrix}$where σ_(z+1) ²(w)=¼[(G_(z) ⁺(w)+G_(z) ⁻(w))²σ_(z) ²(w)+(G_(z)⁺(w)−G_(z) ⁺(w))²σ²] with σ₀ ²=σ².

There is no further post spatial denoising over {circumflex over(R)}(w), which is different from successive robust merging anddifference successive robust merging, which each apply a post spatialdenoising.

FIG. 8 is a flowchart of a method of multi-frame imaging denoising,according to one embodiment.

Referring to FIG. 8, the present system registration of images in asignal at 801.

At 803, the present system extracts patches from the images.

At 805, the present system 2D transforms the extracted patches asdescribed above.

At 807, the present system merges the 2D transformed patches asdescribed above. The method of merging may be one of successive robustmerging, difference successive robust merging, MMSE merging, and U-MMSEmerging, as described above.

At 809, the present system spatially denoises the merged patches.

At 811, the present system 2D inverse transforms and combines thespatially denoised patches.

FIG. 9 is a flowchart of a method of multi-frame imaging denoising,according to one embodiment.

Referring to FIG. 9, the present system determines registration ofimages in a signal at 901.

At 903, the present system extracts patches from the images.

At 905, the present system 2D transforms the extracted patches.

At 907, the present system merges the 2D transformed patches. The methodof merging may be successive pairwise 3D merging.

At 909, the present system 2D inverse transforms and combines mergedpatches.

FIG. 10 is a block diagram of an electronic device in a networkenvironment to which an apparatus and a method of the present disclosureis applied, according to one embodiment.

Referring to FIG. 10, the electronic device 1001 in the networkenvironment 1000 may communicate with an electronic device 1002 via afirst network 1098 (e.g., a short-range wireless communication network),or an electronic device 1004 or a server 1008 via a second network 1099(e.g., a long-range wireless communication network). According to anembodiment, the electronic device 1001 may communicate with theelectronic device 1004 via the server 1008. According to an embodiment,the electronic device 1001 may include a processor 1020, memory 1030, aninput device 1050, a sound output device 1055, a display device 1060, anaudio module 1070, a sensor module 1076, an interface 1077, a hapticmodule 1079, a camera module 1080, a power management module 1088, abattery 1089, a communication module 1090, a subscriber identificationmodule (SIM) 1096, or an antenna module 1097. In some embodiments, atleast one (e.g., the display device 1060 or the camera module 1080) ofthe components may be omitted from the electronic device 1001, or one ormore other components may be added in the electronic device 1001. Insome embodiments, some of the components may be implemented as singleintegrated circuitry. For example, the sensor module 1076 (e.g., afingerprint sensor, an iris sensor, or an illuminance sensor) may beimplemented as embedded in the display device 1060 (e.g., a display).

The processor 1020 may execute, for example, software (e.g., a program1040) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 1001 coupled with theprocessor 1020, and may perform various data processing or computation.

According to one embodiment, as at least part of the data processing orcomputation, the processor 1020 may load a command or data received fromanother component (e.g., the sensor module 1076 or the communicationmodule 1090) in volatile memory 1032, process the command or the datastored in the volatile memory 1032, and store resulting data innon-volatile memory 1034. According to an embodiment, the processor 1020may include a main processor 1021 (e.g., a central processing unit (CPU)or an application processor (AP)), and an auxiliary processor 1023(e.g., a graphics processing unit (GPU), an image signal processor(ISP), a sensor hub processor, or a communication processor (CP)) thatis operable independently from, or in conjunction with, the mainprocessor 1021.

Additionally or non-referencely, the auxiliary processor 1023 may beadapted to consume less power than the main processor 1021, or to bespecific to a specified function. The auxiliary processor 1023 may beimplemented as separate from, or as part of the main processor 1021.

The auxiliary processor 1023 may control at least some of functions orstates related to at least one component (e.g., the display device 1060,the sensor module 1076, or the communication module 1090) among thecomponents of the electronic device 1001, instead of the main processor1021 while the main processor 1021 is in an inactive (e.g., sleep)state, or together with the main processor 1021 while the main processor1021 is in an active state (e.g., executing an application). Accordingto an embodiment, the auxiliary processor 1023 (e.g., an image signalprocessor or a communication processor) may be implemented as part ofanother component (e.g., the camera module 1080 or the communicationmodule 1090) functionally related to the auxiliary processor 1023.

The memory 1030 may store various data used by at least one component(e.g., the processor 1020 or the sensor module 1076) of the electronicdevice 1001. The various data may include, for example, software (e.g.,the program 1040) and input data or output data for a command relatedthereto. The memory 1030 may include the volatile memory 1032 or thenon-volatile memory 1034.

The program 1040 may be stored in the memory 1030 as software, and mayinclude, for example, an operating system (OS) 1042, middleware 1044, oran application 1046.

The input device 1050 may receive a command or data to be used byanother component (e.g., the processor 1020) of the electronic device1001, from the outside (e.g., a user) of the electronic device 1001. Theinput device 1050 may include, e.g., a microphone, a mouse, a keyboard,or a digital pen (e.g., a stylus pen).

The sound output device 1055 may output sound signals to the outside ofthe electronic device 1001. The sound output device 1055 may include,e.g., a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing record, and the receivermay be used for incoming calls. According to an embodiment, the receivermay be implemented as separate from, or as part of the speaker.

The display device 1060 may visually provide information to the outside(e.g., a user) of the electronic device 1001. The display device 1060may include, for example, a display, a hologram device, or a projectorand control circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaydevice 1060 may include touch circuitry adapted to detect a touch, orsensor circuitry (e.g., a pressure sensor) adapted to measure theintensity of force incurred by the touch.

The audio module 1070 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 1070 may obtainthe sound via the input device 1050, or output the sound via the soundoutput device 1055 or a headphone of an external electronic device(e.g., an electronic device 1002) directly (e.g., wiredly) or wirelesslycoupled with the electronic device 1001.

The sensor module 1076 may detect an operational state (e.g., power ortemperature) of the electronic device 1001 or an environmental state(e.g., a state of a user) external to the electronic device 1001, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 1076 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 1077 may support one or more specified protocols to beused for the electronic device 1001 to be coupled with the externalelectronic device (e.g., the electronic device 1002) directly (e.g.,wiredly) or wirelessly. According to an embodiment, the interface 1077may include, for example, a high definition multimedia interface (HDMI),a universal serial bus (USB) interface, a secure digital (SD) cardinterface, or an audio interface.

A connecting terminal 1078 may include a connector via which theelectronic device 1001 may be physically connected with the externalelectronic device (e.g., the electronic device 1002). According to anembodiment, the connecting terminal 1078 may include, for example, aHDMI connector, a USB connector, a SD card connector, or an audioconnector (e.g., a headphone connector).

The haptic module 1079 may convert an electrical signal into amechanical stimulus (e.g., a vibration or a movement) or electricalstimulus which may be recognized by a user via his tactile sensation orkinesthetic sensation. According to an embodiment, the haptic module1079 may include, for example, a motor, a piezoelectric element, or anelectric stimulator.

The camera module 1080 may capture a still image or moving images.According to an embodiment, the camera module 1080 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 1088 may manage power supplied to theelectronic device 1001. According to one embodiment, the powermanagement module 1088 may be implemented as at least part of, forexample, a power management integrated circuit (PMIC).

The battery 1089 may supply power to at least one component of theelectronic device 1001. According to an embodiment, the battery 1089 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 1090 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 1001 and the external electronic device (e.g., theelectronic device 1002, the electronic device 1004, or the server 1008)and performing communication via the established communication channel.The communication module 1090 may include one or more communicationprocessors that are operable independently from the processor 1020(e.g., the application processor (AP)) and supports a direct (e.g.,wired) communication or a wireless communication. According to anembodiment, the communication module 1090 may include a wirelesscommunication module 1092 (e.g., a cellular communication module, ashort-range wireless communication module, or a global navigationsatellite system (GNSS) communication module) or a wired communicationmodule 1094 (e.g., a local area network (LAN) communication module or apower line communication (PLC) module). A corresponding one of thesecommunication modules may communicate with the external electronicdevice via the first network 1098 (e.g., a short-range communicationnetwork, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, orinfrared data association (IrDA)) or the second network 1099 (e.g., along-range communication network, such as a cellular network, theInternet, or a computer network (e.g., LAN or wide area network (WAN)).These various types of communication modules may be implemented as asingle component (e.g., a single chip), or may be implemented as multicomponents (e.g., multi chips) separate from each other. The wirelesscommunication module 1092 may identify and authenticate the electronicdevice 1001 in a communication network, such as the first network 1098or the second network 1099, using subscriber information (e.g.,international mobile subscriber identity (IMSI)) stored in thesubscriber identification module 1096.

The antenna module 1097 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 1001. According to an embodiment, the antenna module1097 may include an antenna including a radiating element composed of aconductive material or a conductive pattern formed in or on a substrate(e.g., PCB). According to an embodiment, the antenna module 1097 mayinclude a plurality of antennas. In such a case, at least one antennaappropriate for a communication scheme used in the communicationnetwork, such as the first network 1098 or the second network 1099, maybe selected, for example, by the communication module 1090 (e.g., thewireless communication module 1092) from the plurality of antennas. Thesignal or the power may then be transmitted or received between thecommunication module 1090 and the external electronic device via theselected at least one antenna. According to an embodiment, anothercomponent (e.g., a radio frequency integrated circuit (RFIC)) other thanthe radiating element may be additionally formed as part of the antennamodule 1097.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 1001 and the external electronicdevice 1004 via the server 1008 coupled with the second network 1099.Each of the electronic devices 1002 and 1004 may be a device of a sametype as, or a different type, from the electronic device 1001. Accordingto an embodiment, all or some of operations to be executed at theelectronic device 1001 may be executed at one or more of the externalelectronic devices 1002, 1004, or 1008. For example, if the electronicdevice 1001 should perform a function or a service automatically, or inresponse to a request from a user or another device, the electronicdevice 1001, instead of, or in addition to, executing the function orthe service, may request the one or more external electronic devices toperform at least part of the function or the service. The one or moreexternal electronic devices receiving the request may perform the atleast part of the function or the service requested, or an additionalfunction or an additional service related to the request, and transferan outcome of the performing to the electronic device 1001. Theelectronic device 1001 may provide the outcome, with or without furtherprocessing of the outcome, as at least part of a reply to the request.To that end, a cloud computing, distributed computing, or client-servercomputing technology may be used, for example.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smartphone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According toan embodiment of the disclosure, the electronic devices are not limitedto those described above.

Various embodiments of the present disclosure and the terms used hereinare not intended to limit the technological features of presentdisclosure to particular embodiments and include various changes,equivalents, or replacements for a corresponding embodiment.

With regard to the description of the drawings, similar referencenumerals may be used to refer to similar or related elements.

A singular form of a noun corresponding to an item may include one ormore of the things, unless the relevant context clearly indicatesotherwise.

As used herein, each of such phrases as “A or B,” “at least one of A andB,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, andC,” and “at least one of A, B, or C,” may include any one of, or allpossible combinations of the items enumerated together in acorresponding one of the phrases.

Terms such as “1^(st)” and “2nd” may be used to simply distinguish acorresponding component from another, and do not limit the components inother aspects (e.g., importance or order). If an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “, coupled to” “with connected,”or “connected to” another element (e.g., a second element), the firstelement may be coupled with the second element directly (e.g., wiredly),wirelessly, or via a third element.

As used herein, the term “module” may include a unit implemented inhardware, software, or firmware, and may interchangeably be used withother terms, for example, “logic,” “logic block,” “part,” or“circuitry”. A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. Forexample, according to an embodiment, the module may be implemented in aform of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 1040) including one or more instructions that arestored in a storage medium (e.g., internal memory 1036 or externalmemory 1038) that is readable by a machine (e.g., the electronic device1001). For example, a processor (e.g., the processor 1020) of themachine (e.g., the electronic device 1001) may invoke at least one ofthe one or more instructions stored in the storage medium, and executeit, with or without using one or more other components under the controlof the processor. This allows the machine to be operated to perform atleast one function according to the at least one instruction invoked.The one or more instructions may include a code generated by a complieror a code executable by an interpreter. The machine-readable storagemedium may be provided in the form of a non-transitory storage medium.Wherein, the term “non-transitory” simply means that the storage mediumis a tangible device, and does not include a signal (e.g., anelectromagnetic wave), but this term does not differentiate betweenwhere data is semi-permanently stored in the storage medium and wherethe data is temporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., PlayStore™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to various embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities. According to various embodiments, one or more ofthe above-described components may be omitted, or one or more othercomponents may be added. Non-referencely or additionally, a plurality ofcomponents (e.g., modules or programs) may be integrated into a singlecomponent. In such a case, according to various embodiments, theintegrated component may still perform one or more functions of each ofthe plurality of components in the same or similar manner as they areperformed by a corresponding one of the plurality of components beforethe integration. According to various embodiments, operations performedby the module, the program, or another component may be carried outsequentially, in parallel, repeatedly, or heuristically, or one or moreof the operations may be executed in a different order or omitted, orone or more other operations may be added.

FIG. 11 is a block diagram of the program 1040 according to oneembodiment.

Referring to FIG. 11, the program 1040 may include an OS 1042 to controlone or more resources of the electronic device 1001, middleware 1044, oran application 1046 executable in the OS 1042. The OS 1042 may include,for example, Android®, iOS®, Windows®, Symbian®, Tizen®, or Bada™. Atleast part of the program 1040, for example, may be pre-loaded on theelectronic device 1001 during manufacture, or may be downloaded from orupdated by an external electronic device (e.g., the electronic device1002 or 1004, or the server 1008) during use by a user.

The OS 1042 may control management (e.g., allocating or deallocation) ofone or more system resources (e.g., process, memory, or power source) ofthe electronic device 1001. The OS 1042, additionally ornon-referencely, may include one or more driver programs to drive otherhardware devices of the electronic device 1001, for example, the inputdevice 1050, the sound output device 1055, the display device 1060, theaudio module 1070, the sensor module 1076, the interface 1077, thehaptic module 1079, the camera module 1080, the power management module1088, the battery 1089, the communication module 1090, the subscriberidentification module 1096, or the antenna module 1097.

The middleware 1044 may provide various functions to the application1046 such that a function or information provided from one or moreresources of the electronic device 1001 may be used by the application1046. The middleware 1044 may include, for example, an applicationmanager 1101, a window manager 1103, a multimedia manager 1105, aresource manager 1107, a power manager 1109, a database manager 1111, apackage manager 1113, a connectivity manager 1115, a notificationmanager 1117, a location manager 1119, a graphic manager 1121, asecurity manager 1123, a telephony manager 1125, or a voice recognitionmanager 1127.

The application manager 1101, for example, may manage the life cycle ofthe application 1046. The window manager 1103, for example, may manageone or more graphical user interface (GUI) resources that are used on ascreen. The multimedia manager 1105, for example, may identify one ormore formats to be used to play media files, and may encode or decode acorresponding one of the media files using a codec appropriate for acorresponding format selected from the one or more formats. The resourcemanager 1107, for example, may manage the source code of the application1046 or a memory space of the memory 1030. The power manager 1109, forexample, may manage the capacity, temperature, or power of the battery1089, and determine or provide related information to be used for theoperation of the electronic device 1001 based at least in part oncorresponding information of the capacity, temperature, or power of thebattery 1089. According to one embodiment, the power manager 1109 mayinteroperate with a basic input/output system (BIOS) of the electronicdevice 1001.

The database manager 1111, for example, may generate, search, or changea database to be used by the application 1046. The package manager 1113,for example, may manage installation or update of an application that isdistributed in the form of a package file. The connectivity manager1115, for example, may manage a wireless connection or a directconnection between the electronic device 1001 and the externalelectronic device. The notification manager 1117, for example, mayprovide a function to notify a user of an occurrence of a specifiedevent (e.g., an incoming call, message, or alert). The location manager1119, for example, may manage locational information on the electronicdevice 1001. The graphic manager 1121, for example, may manage one ormore graphic effects to be offered to a user or a user interface relatedto the one or more graphic effects.

The security manager 1123, for example, may provide system security oruser authentication. The telephony manager 1125, for example, may managea voice call function or a video call function provided by theelectronic device 1001. The voice recognition manager 1127, for example,may transmit a user's voice data to the server 1008, and receive, fromthe server 1008, a command corresponding to a function to be executed onthe electronic device 1001 based at least in part on the voice data, ortext data converted based at least in part on the voice data. Accordingto one embodiment, the middleware 1044 may dynamically delete someexisting components or add new components. According to one embodiment,at least part of the middleware 1044 may be included as part of the OS1042 or may be implemented in other software separate from the OS 1042.

The application 1046 may include, for example, a home application 1151,a dialer application 1153, a short message service (SMS)/multimediamessaging service (MMS) application 1155, an instant message (IM)application 1157, a browser application 1159, a camera application 1161,an alarm application 1163, a contact application 1165, a voicerecognition application 1167, an email application 1169, a calendarapplication 1171, a media player application 1173, an album application1175, a watch application 1177, a health application 1179 (e.g., formeasuring the degree of workout or biometric information, such as bloodsugar), or an environmental information application 1181 (e.g., formeasuring air pressure, humidity, or temperature information). Accordingto one embodiment, the application 1046 may further include aninformation exchanging application that is capable of supportinginformation exchange between the electronic device 1001 and the externalelectronic device. The information exchange application, for example,may include a notification relay application adapted to transferdesignated information (e.g., a call, a message, or an alert) to theexternal electronic device or a device management application adapted tomanage the external electronic device. The notification relayapplication may transfer notification information corresponding to anoccurrence of a specified event (e.g., receipt of an email) at anotherapplication (e.g., the email application 1169) of the electronic device1001 to the external electronic device. Additionally or non-referencely,the notification relay application may receive notification informationfrom the external electronic device and provide the notificationinformation to a user of the electronic device 1001.

The device management application may control the power (e.g., turn-onor turn-off) or the function (e.g., adjustment of brightness,resolution, or focus) of the external electronic device or somecomponent thereof (e.g., a display device or a camera module of theexternal electronic device). The device management application,additionally or non-referencely, may support installation, delete, orupdate of an application running on the external electronic device.

Although certain embodiments of the present disclosure have beendescribed in the detailed description of the present disclosure, thepresent disclosure may be modified in various forms without departingfrom the scope of the present disclosure. Thus, the scope of the presentdisclosure shall not be determined merely based on the describedembodiments, but rather determined based on the accompanying claims andequivalents thereto.

What is claimed is:
 1. An apparatus, comprising: a first subtractorincluding a first input to receive a frame of an image, a second input,and an output; a first Wiener filter including at least one inputconnected to the output of the first subtractor, and configured tofilter the frame of the image and output a filtered frame; a summationfunction block or a first adder including a first input connected to anoutput of the first Wiener filter, a second input connected to the firstinput of the first subtractor or an output of a second Wiener filter,and an output; a multiplier including a first input connected to theoutput of the summation function block or the first adder, a secondinput for receiving a predetermined fractional value less than one, andan output; and a frame delayer including a first input connected to theoutput of the multiplier, and an output, and configured to feedback, tothe first Wiener filter, an estimate of the image that is based on thefiltered frame, wherein the second input of the subtractor is the outputof the frame delayer or a reference frame, and wherein the first Wienerfilter is further configured to filter each subsequent frame of theimage based on a corresponding estimate of the image that is based on apreviously filtered frame.
 2. The apparatus of claim 1, furthercomprising: the first subtractor including the second input connected tothe output of the frame delayer, and the output connected to a firstinput and a second input of the first Wiener filter; the summationfunction block including the second input connected to the first inputof the first subtractor; and the multiplier including the first inputconnected to the output of the summation function block.
 3. Theapparatus of claim 2, further comprising: an image registrator includingan input and an output; a patch extractor including an input connectedto the output of the image registrator and an output; a two dimensional(2D) transformer including an input connected to the output of the patchextractor and an output connected to the first input of the firstsubtractor; a spatial denoiser including an input connected to theoutput of the summation function block and an output; and a 2D inversetransformer and patch combiner including an input connected to theoutput of the spatial denoiser and an output.
 4. The apparatus of claim1, further comprising: the first subtractor including the second inputto receive the reference frame, and an output connected to a first inputof the first Wiener filter; the summation function block including thesecond input connected to the first input of the first subtractor; andthe multiplier including the first input connected to the output of thesummation function block; and a second subtractor including a firstinput connected to the output of the frame delayer, a second inputconnected to the first input of the first subtractor, and an outputconnected to a second input of the first Wiener filter.
 5. The apparatusof claim 4, further comprising: an image registrator including an inputand an output; a patch extractor including an input connected to theoutput of the image registrator and an output; a two dimensional (2D)transformer including an input connected to the output of the patchextractor and an output connected to the first input of the firstsubtractor; a spatial denoiser including an input connected to theoutput of the summation function block and an output; and a 2D inversetransformer and patch combiner including an input connected to theoutput of the spatial denoiser and an output.
 6. The apparatus of claim1, further comprising: the first subtractor including the second inputconnected to the output of the frame delayer, and the output connectedto a first input and a second input of the first Wiener filter; a secondadder including a first input connected to the first input of the firstsubtractor, a second input connected to the output of the frame delayer,and an output; the second Wiener filter including a first input and asecond input connected to the output of the second adder, and an output;the first adder including the second input connected to the output ofthe second Wiener filter; and the multiplier including the first inputconnected to the output of the first adder.
 7. The apparatus of claim 6,further comprising: an image registrator including an input and anoutput; a patch extractor including an input connected to the output ofthe image registrator and an output; a two dimensional (2D) transformerincluding an input connected to the output of the patch extractor and anoutput connected to the first input of the first subtractor; a spatialdenoiser including an input connected to the output of the first adderand an output; and a 2D inverse transformer and patch combiner includingan input connected to the output of the spatial denoiser and an output.8. A method, comprising: subtracting, by a first subtractor, a firstinput of a frame of an image from a second input; filtering, by a firstWiener filter, the frame of the image to output a filtered frame;summing, by a summation function block or a first adder, an output ofthe first Weiner filter, and the first input of the subtractor or anoutput of a second Wiener filter; multiplying, by a multiplier, apredetermined fractional value less than one and an output of thesummation function block or the first adder; inputting a result of themultiplication to a frame delayer; feeding back, by the frame delayer,to the first Wiener filter, an estimate of the image that is based onthe filtered frame; and filtering, by the first Wiener filter, eachsubsequent frame of the image based on a corresponding estimate of theimage that is based on a previously filtered frame, wherein the secondinput of the first subtractor is an output of the frame delayer or areference frame.
 9. The method of claim 8, further comprising:subtracting, by the first subtractor, the frame of the image from theoutput of the frame delayer, wherein the summing comprises summing, bythe summation function block, the output of the first Wiener filter andthe first input of the first subtractor, and wherein the multiplyingcomprises multiplying, by the multiplier, the output of the summationfunction block and the predetermined fractional value less than one. 10.The method of claim 9, further comprising: determining, by an imageregistrator, a registration of images in a signal; extracting, by apatch extractor, patches from the images; transforming, by a twodimensional (2D) transformer, an output of the patch extractor;spatially denoising, by a spatial denoiser, the output of the summationfunction block; and transforming, by a 2D inverse transformer and patchcombiner, the output of the spatial denoiser; and combining, by the 2Dinverse transformer and patch combiner, patches from an output of the 2Dinverse transformer and patch combiner.
 11. The method of claim 8,further comprising: subtracting, by the first subtractor, the frame ofthe image from the reference frame, wherein the summing comprisessumming, by the summation function block, the output of the first Wienerfilter and the frame of the image received by the first subtractor, andwherein the multiplying comprises multiplying, by the multiplier, theoutput of the summation function block and the predetermined fractionalvalue less than one; subtracting, by a second subtractor, the frame ofthe image from the output of the frame delayer; and providing an outputof the second subtractor to the first Wiener filter.
 12. The method ofclaim 11, further comprising: determining, by an image registrator, aregistration of images in a signal; extracting, by a patch extractor,patches from the images; transforming, by a two dimensional (2D)transformer, an output of the patch extractor; providing the output of2D transformer to the first subtractor; spatially denoising, by aspatial denoiser, the output of the summation function block;transforming, by a 2D inverse transformer and patch combiner, the outputof the spatial denoiser; and combining, by the 2D inverse transformerand patch combiner, patches from an output of the 2D inverse transformerand patch combiner.
 13. The method of claim 8, further comprising:subtracting, by a first subtractor, the frame of the image from theoutput of the frame delayer; providing a result of the subtraction tothe first Wiener filter; adding, by a second adder, the frame of theimage and the output of the frame delayer; filtering, by the secondWiener filter, the output of the second adder; wherein the summingcomprises adding, by the first adder, the output of the first Wienerfilter and the output of the second Wiener filter, and wherein themultiplying comprises multiplying, by the multiplier, the output of thefirst adder and the predetermined fractional value less than one. 14.The method of claim 13, further comprising: determining, by an imageregistrator, a registration of images in a signal; extracting, by apatch extractor, patches from the images; transforming, by a twodimensional (2D) transformer, an output of the patch extractor;providing an output of the 2D transformer to the first subtractor;spatially denoising, by a spatial denoiser, the output of the firstadder; transforming, by a 2D inverse transformer and patch combiner, anoutput of the spatial denoiser; and combining, by the 2D inversetransformer and patch combiner, patches from an output of the 2D inversetransformer and patch combiner.